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CLAIMS 

1 1 . An interactive multimedia delivery system dynamically linking contextual information 

2 with multimedia documents, said system retrieving said contextual information by 

3 searching an ontology and one or more databases over a network, said ontology 

4 comprising one or more nodes, said system comprising: 

5 a. a learning data preparation component accessing mappings of annotations in said 

6 ontology and fusing annotations mapped in each of said nodes to form learning 
O 7 instances; 

8 b. an intelligent inverted index creating a data structure based on the following 

[ij 9 calculated statistics for said learning instances: term frequency (tf), inverse document 

,30 frequency (idf), and contribution frequency (cf); 

13.1 c. a retriever receiving a request for new annotations associated with multimedia 

pl2 documents, said retriever utilizing said inverted index to retrieve and rank most 

Jj_3 relevant nodes for said received new annotations, said ranking determined based 

14 upon a weight, wty, contributed to a particular node in said ontology by the 

15 occurrence of a word i in a learning instance j; 

16 d. an information retriever extracting information related to said requested annotations 

1 7 from said most relevant nodes and said one or more databases over said network, and 

18 e. a contextual information linker linking multimedia content with said extracted 

19 information. 

l 2. An interactive multimedia delivery system dynamically linking contextual information 
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with multimedia documents, as per claim 1, wherein said weight wty is given by: 
wt fJ = (0.4 + 0.6 x Normalized _ tf~ x idfj )xwt_cf 

An interactive multimedia delivery system dynamically linking contextual information 
with multimedia documents, as per claim 1, wherein said multimedia documents 
comprises audio, text, graphics, video documents. 

An interactive multimedia delivery system dynamically linking contextual information 
with multimedia documents, as per claim 1, wherein said annotations are accessible via 
any of the following devices: an interactive television, a computer, a portable computer, a 
handheld device, or a telephone. 

An interactive multimedia delivery system dynamically linking contextual information 
with multimedia documents, as per claim 1, wherein said network is any of the 
following: wide area network (WAN), local area network (LAN), wireless network, the 
telephony network, or the Internet. 

An interactive multimedia delivery system dynamically linking contextual information 
with multimedia documents, as per claim 1, said learning data preparation further 
comprising: 

a tokenizer, which tokenizes said learning instances; 

a stemmer which stems said tokenized learning instances, and 

a stop-word-remover, which removes stop words from said stemmed tokenized learning 
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instances. 

7. A method for searching an ontology of mapped multimedia annotations for appropriate 
annotations for one more multimedia documents, said ontology comprising one or more 
nodes, said method comprising the steps of: 

a. receiving a request for searching and extracting one or more annotations related to 
said multimedia documents from said ontology; 

b. identifying nodes in said ontology that are relevant to said multimedia documents, 
said nodes further comprising fused learning instances formed by fusing annotations 
in each of said nodes, said identification based upon using special statistics including 
term frequency, inverse document frequency and contribution frequency; 

c. extracting information from said identified relevant nodes, and 

d. dynamically linking said extracted information with said multimedia documents. 

8. A method for searching an ontology of mapped multimedia annotations for appropriate 
annotations for one more multimedia documents, as per claim 7, wherein said multimedia 
documents comprises audio, text, graphics, video documents. 

9. A method for searching an ontology of mapped multimedia annotations for appropriate 
annotations for one more multimedia documents, as per claim 7, wherein said 
annotations are accessible via any of the following devices: an interactive television, a 
computer, a portable computer, a handheld device. 
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1 10. A method for searching an ontology of mapped multimedia annotations for appropriate 

2 annotations for one more multimedia documents, as per claim 7, said method further 

3 comprising; 

4 tokenizing said learning instances; 

5 stemming said tokenized learning instances, and 

6 removing stop words from said stemmed tokenized learning instances. 

1 11. A method for retrieving contextual information by searching an ontology and one or 

2 more databases, said method comprising: 

3 receiving a request for contextual information; 

4 retrieving from an ontology, with automatically mapped annotations, said requested 

5 contextual information using information retrieval statistics; 

6 retrieving said requested contextual information from one or more databases, and 

7 rendering an integrated presentation comprising audio, video, or graphics and said 

8 retrieved contextual information. 

1 12. A method for retrieving contextual information by searching an ontology and one or 

2 more databases, as per claim 11, wherein said information retrieval statistics include 

3 calculating the following parameters: 

log(C+0.5) 

4 1) Normalized tf i} = 0.4 + 0.6 x- — - — J - — - — — 

log(max_^.+l) 
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8( ^ r) 

2) idf. = ^~ 

\og(N) 

3) - (0.5 + J) 0.o- r ^ ? ) 

4) w t = (0A + 0.6x Normalized _tf u xidf J )xwt_cf 

A method for retrieving contextual information by searching an ontology and one or 
more databases, as per claim 11, wherein said information retrieval statistic further 
comprises calculating a weight contributed by a particular category in said ontology by a 
occurrence of word i in a learning vector y, said weight given by: 
w t.. = (0.4 + 0.6 x Normalized _ tf-xidfj ) xwt _ cf 

A method for retrieving contextual information by searching an ontology and one or 
more databases, as per claim 11, wherein said weight further depends on a contribution 
frequency, said contribution frequency given by the number of annotations (that 
comprises said learning instance) in which said word i appears. 

A method for retrieving contextual information by searching an ontology and one or 
more databases, as per claim 1 1, wherein said annotations are retrieved from any of the 
following sources: text documents, message boards, chat rooms, product descriptions, 
and multimedia documents comprising audio, video, images, and graphics in various 
formats. 
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1 16, A method for retrieving contextual information by searching an ontology and one or 

2 more databases, as per claim 11, wherein said annotations are viewable via any of the 

3 following devices : an interactive television, a computer, or a handheld device, connected 

4 to the Internet, a cable system, or a wireless network. 

1 17. A method for retrieving contextual information by searching an ontology and one or 

2 more databases, as per claim 11, wherein said databases are located on a network. 

1 18. A method for retrieving contextual information by searching an ontology and one or 

2 more databases, as per claim 1 7, wherein said network is any of the following: local area 

3 network (LAN), wide area network (WAN), wireless network, world wide web (WWW), 

4 or Internet. 

1 19. A system for retrieving contextual information by searching for a selected multimedia 

2 representation, said system comprising: 

3 a server, said server receiving requests for contextual information for a selected 

4 multimedia representation; 

5 one or more databases associated with said server, 

6 wherein said server retrieves both from its own ontology, said ontology having 

7 automatically mapped annotations, and from said one or more databases said requested 

8 contextual information, and renders said retrieved information as an integrated 

9 presentation comprising said multimedia and said retrieved contextual information. 
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1 20. A system for retrieving contextual information by searching for a selected multimedia 

2 representation, as per claim 19, wherein said information retrieval statistics includes 

3 calculating the following parameters: 

log(C+0.5) 

4 1) Normalized tf B = 0.4 + 0.6 x- — — - 

- " log(max_// / . + 1) 

3) ,t_ cf = (o.5 + ^) (i.o-^r) 

7 4) wt = (0.4 + 0.6x Normalized _tf~xidfj)xwt _cf 

1 21 . A system for retrieving contextual information by searching for a selected multimedia 

2 representation, as per claim 19, wherein said information retrieval statistic further 

3 comprises calculating a weight contributed by a particular category in said ontology by a 

4 occurrence of word i in a learning vector j 9 said weight given by: 

5 w t = (0.4 + 0.6 x Normalized tfyXidfj ) xwt _ c/ 

1 22. A system for retrieving contextual information by searching for a selected multimedia 

2 representation, as per claim 21, wherein said weight further depends on a contribution 

3 frequency, said contribution frequency given by the number of annotations (that 

4 comprises said learning instance) in which said word i appears. 
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1 23. A system for retrieving contextual information by searching for a selected multimedia 

2 representation, as per claim 19, wherein said contextual information are retrieved from 

3 any of the following sources: text documents, message boards, chat rooms, product 

4 descriptions, and multimedia documents comprising audio, video, images, and graphics 

5 in various formats. 

1 24. A system for retrieving contextual information by searching for a selected multimedia 

2 representation, as per claim 1 9, wherein said contextual information is accessible via any 

3 of the following devices: an interactive television, a computer, or a handheld device, 

4 connected to the Internet, a cable system, or a wireless network. 

1 25. A system for retrieving contextual information by searching for a selected multimedia 

2 representation, as per claim 19, wherein said databases are located on a network. 

1 26. A system for retrieving contextual information by searching for a selected multimedia 

2 representation, as per claim 25, wherein said network is any of the following: local area 

3 network (LAN), wide area network (WAN), wireless network, world wide web (WWW), 

4 or Internet. 

1 27 . A method for automatically mapping annotations to ontologies, said method comprising 

2 the steps of: 

3 extracting annotations from a multimedia document segment; 

4 mapping said extracted multimedia document segment to an appropriate node in said 
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5 ontology; 

s comparing to other related content mapped to said appropriate node, and 

7 integrating said related content with said extracted multimedia document segment. 

1 28 . A method for automatically mapping annotations to ontologies, as per claim 27, wherein 

2 pre-certification of said related content is required before said integration step. 

1 29. A method for automatically mapping annotations to ontologies, as per claim 27, wherein 

2 said step of integration is accomplished via dynamic content linking. 

1 30. A method for automatically mapping annotations to ontologies, as per claim 27, wherein 

2 said annotations are retrieved from any of the following sources: text documents, 

3 message boards, chat rooms, product descriptions, and multimedia documents 

4 comprising audio, video, images, and graphics in various formats. 

1 31. A method for automatically mapping annotations to ontologies, as per claim 27, wherein 

2 said annotations are accessible via any of the following devices: an interactive television, 

3 a computer, or a handheld device, connected to the Internet, a cable system, or a wireless 

4 network. 

1 32. An article of manufacture comprising a computer usable medium having computer 

2 readable program code embodied therein which searches an ontology of mapped 

3 multimedia annotations for appropriate annotations for one more multimedia documents, 
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4 said ontology comprising one or more nodes, said article comprising: 

5 a. computer readable program code receiving a request for searching and extracting one 

6 or more annotations related to said multimedia documents from said ontology; 

7 b. computer readable program code identifying nodes in said ontology that are relevant 

8 to said multimedia documents, said nodes further comprising fused learning instances 

9 formed by fusing annotations in each of said nodes, said identification based upon 
io using special statistics including term frequency, inverse document frequency and 

|3l i contribution frequency; 

;l| 2 c. computer readable program code extracting information from said identified relevant 

,y.3 nodes, and 

: a 4 d. computer readable program code dynamically linking said extracted information with 

d s said multimedia documents. 
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